- name: kubernetes.node.chrony
  rules:
  - record: node_ntp_offset_seconds:abs
    expr: abs(node_ntp_offset_seconds)

  - alert: NTPDaemonOnNodeDoesNotSynchronizeTime
    expr: (min by (node) (node_timex_sync_status{job="node-exporter"})) == 0
    for: 30m
    labels:
      severity_level: "5"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: markdown
      description: |
        1. check if Chrony pod is running on the node by executing the following command:
           * 'kubectl -n d8-chrony --field-selector spec.nodeName="{{$labels.node}}" get pods'
        2. check the Chrony daemon's status by executing the following command:
           * 'kubectl -n d8-chrony exec <POD_NAME> -- /opt/chrony-static/bin/chronyc sources'
        3. Correct the time synchronization problems:
           * correct network problems:
             - provide availability to upstream time synchronization servers defined in the module [configuration](https://deckhouse.io/documentation/v1/modules/470-chrony/configuration.html);
             - eliminate large packet loss and excessive latency to upstream time synchronization servers.
           * Modify NTP servers list defined in the module [configuration](https://deckhouse.io/documentation/v1/modules/470-chrony/configuration.html).
      summary: NTP daemon on node {{$labels.node}} have not synchronized time for too long.

  - alert: NodeTimeOutOfSync
    expr: abs(node_ntp_offset_seconds:abs - (node_ntp_rtt_seconds/2)) > 0.05
    for: 30m
    labels:
      severity_level: "5"
      tier: cluster
    annotations:
      plk_protocol_version: "1"
      plk_markup_format: markdown
      description: |
        Node's {{$labels.node}} time is out of sync from ntp server by {{ $value }} seconds.
      summary: Node's {{$labels.node}} clock is drifting.
